Vigenère cipher is an ancient elementary method that uses a series of Caesar shifting for encrypting plaintext to ciphertext protecting it from adversaries. However, a frequency analysis attack is vulnerable in this type of cipher technique. This paper aims to enhance the security of the Vigenère cipher in a hybrid cryptosystem using a randomized approach including pseudorandom substitution, rearranging, padding, etc. In this approach, a complex key generation algorithm is used that generates a pair of subkeys from an input key. One of the subkeys, Subkey1 is used to generate an intermediate ciphertext. The other key, Subkey2 is used to further scramble the ciphertext. The slightest change in the input symmetric key produces an entirely different key pair. Hence, different encrypted results. It also increases the ranges of characters that Vigenère cipher can encrypt by including all ASCII and extended ASCII. Finally, the symmetric key is encrypted with the public key of RSA solving the key distribution problem of the symmetric cipher. This enhanced and modified Vigenère cipher in the hybrid cryptosystem overcomes all limitations of classical cipher and acts as a bridge between classical and modern cryptography.

Caesar Cipher Vigenère Cipher Encryption Decryption Key Plaintext Ciphertext Cryptanalysis RSA Security Symmetric Asymmetric