Encrypted File System in Minix

Posted on 2005-04-27
Last Modified: 2013-12-04

I am trying to develop an encrypted file system in minix(a simulated OS like UNIX). The encryption mechanism uses AES algorithm. Right now I am having trouble with getting key from command line and encrypt the file using that key.
I have written this demo program which takes filename and key as user input. Then shows the encrypted and decrypted result. It encrypts data block by block. Here is my demo.c built in UNIX and then compiled and sunread to minix.

/* chmod - Change file modes                            Author: V. Archer */

/* Copyright 1991 by Vincent Archer
 *      You may freely redistribute this software, in source or binary
 *      form, provided that you do not alter this copyright mention in any
 *      way.

#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <errno.h>
#include <limits.h>
#include <stdlib.h>  
#include <string.h>
#include <unistd.h>
#include <minix/minlib.h>
#include <stdio.h>
#include <assert.h>
#include <rijndael-api-fst.h>
#define MAXBC 256
#ifndef S_ISLNK
#define S_ISLNK(mode)   0
#define lstat           stat

#ifdef S_ISVTX

  /* Common variables */
char *symbolic;
mode_t new_mode, u_mask;
int rflag, errors;
struct stat st;
char path[PATH_MAX + 1];

/* Main module. The single option possible (-R) does not warrant a call to
 * the getopt() stuff.
int main(argc, argv)
int argc;
char *argv[];
        AES_BYTE inBlock[4*MAXBC], outBlock[4*MAXBC],outBlock1[4*MAXBC], binKey[4*MAXKC];
        AES_BYTE direction,keyMaterial[32];
        keyInstance keyInst;
        cipherInstance cipherInst;
        FILE *fp,*fp2;
        int i,j,k,keyLength,blockLength;
        char *test11, *test22,*initBlock,*initKey,*initKey2;
        char test1[1024], test2[1024];
        int x;
        int flag = 0;
        direction =0;
        //int l=0;
        //char *buf1,*buf2;

        //char data[20]="DATATOENCRPT";
        initKey   = "abcdeabcdeabcdeaabcdeabcdeabcdea";
        //initKey2  = "12233344445555566666677777778888";
        keyLength = 128;
        blockLength = 8192;

        if(argc != 3)
                printf("Enter a valid filename and an encryption key\n");
        printf("Filename : %s \nKey :  %s\n",argv[1],argv[2]);
        fp = fopen(argv[1],"r");
        //fp2 = fopen(argv[1],"a");
        if(fp == NULL)
                printf(" file not found \n");
                initKey = argv[2];
          printf("InitKey : %s\n",initKey);
          //printf("Enter a key to encrypt : ");
          //    scanf("%c",initKey[l]);
          //initKey[l] = '\0';
          printf("file opened, going on\n");
          while(fgets(test1,1024,fp) != NULL || flag != 1 )
              direction = 0;
              printf("copied some characters and am going to encrypt");
              flag = 1;
              printf("\nCharacters copied : %s\n", test1);
              //printf("iteration number: %d\n",++l);
              test1[strlen(test1) -1] = '\0';
              test11 = test1;
              j = 4*MAXBC;
                inBlock[i] = test11[i];
                keyMaterial[i] = initKey[i];
              printf("\n key is := ");
              //keyInst.blockLen = blockLength;
              makeKey(&keyInst, direction, keyLength, keyMaterial);
              printf("Made key, continuing\n");
              //cipherInst.blockLen = blockLength;
              cipherInit (&cipherInst, MODE_ECB, NULL);
              printf("Made cipher, continuing\n");
              if (direction == DIR_ENCRYPT)
                  printf("before memcpy inblock => %x and outblock => %x\n\n",inBlock,outBlock);
                  memcpy (outBlock, inBlock, blockLength/8);
                  printf("Memory copy done\n");
                  blockEncrypt(&cipherInst, &keyInst, inBlock, blockLength, outBlock);
                  printf("\n finish encrypt inblock is => %x and outblock is => %s\n",inBlock,outBlock);
                  //   buf2[i] = (char)outBlock[i];
              direction = 1;
              keyMaterial[keyLength/4] = 0;
              //keyInst.blockLen = blockLength;
              makeKey(&keyInst, direction, keyLength, keyMaterial);
              //cipherInst.blockLen = blockLength;
              cipherInit (&cipherInst, MODE_ECB, NULL);
              printf("outblock => %x and outblock1 = %x \n",outBlock,outBlock1);
              blockDecrypt(&cipherInst, &keyInst, outBlock, blockLength, outBlock1);
              printf("\n decrypt outBlock is => %x and outBlock1 => %s\n",outBlock,outBlock1);
                //printf("\nContents of encrypted file \n");
                //while(fgets(buf1,1024,fp2) != NULL)
                //      printf("%s\n",buf1);

When I give a 32-bit key in numeric such as 12345678901234567890123456789045 it works fine but when I call demo with a alphanumeric key such as abcdefghijklmnopqrstuvwxyz123456 it shows this error

! demo abc.txt abcdefghijklmnopqrstuvwxyz123456
Filename : abc.txt
Key :  abcdefghijklmnopqrstuvwxyz123456
InitKey : abcdefghijklmnopqrstuvwxyz123456
file opened, going on
copied some characters and am going to encrypt
Characters copied : this is a new file

 key is := abcdefghijklmnopqrstuvwxyz123456
Made key, continuing
Made cipher, continuing
before memcpy inblock => 1fffab0 and outblock => 1fff6b0

Memory copy done
SunOS sig 11 in 9 (demo, sp 0x1ffe6d8) accessing 0x2300000 at 0x13de8
Memory fault - core dumped

Can anybody please help me to fix this error. Really need it. Any questions are welcome.
Question by:jamogle
    LVL 51

    Accepted Solution

    the problem ist most likely in
       blockEncrypt(&cipherInst, &keyInst, inBlock, blockLength, outBlock);

    you pass some arguments as reference, others as value, are you shure that this is as the declaration of blockEncrypt89 expects it?

    Author Comment

    I guess the arguments are passed correctly. Here is the block encrypt fn.

    int blockEncrypt(cipherInstance *cipher, keyInstance *key,
                AES_BYTE *input, int inputLen, AES_BYTE *outBuffer) {
          int i, k, t, numBlocks;
          u8 block[16], *iv;

          if (cipher == NULL ||
                key == NULL ||
                key->direction == DIR_DECRYPT) {
                return BAD_CIPHER_STATE;
          if (input == NULL || inputLen <= 0) {
                return 0; /* nothing to do */

          numBlocks = inputLen/128;
          switch (cipher->mode) {
          case MODE_ECB:
                for (i = numBlocks; i > 0; i--) {
                      rijndaelEncrypt(key->rk, key->Nr, input, outBuffer);
                      input += 16;
                      outBuffer += 16;
          case MODE_CBC:
                iv = cipher->IV;
                for (i = numBlocks; i > 0; i--) {
                      ((u32*)block)[0] = ((u32*)input)[0] ^ ((u32*)iv)[0];
                      ((u32*)block)[1] = ((u32*)input)[1] ^ ((u32*)iv)[1];
                      ((u32*)block)[2] = ((u32*)input)[2] ^ ((u32*)iv)[2];
                      ((u32*)block)[3] = ((u32*)input)[3] ^ ((u32*)iv)[3];
                      rijndaelEncrypt(key->rk, key->Nr, block, outBuffer);
                      iv = outBuffer;
                      input += 16;
                      outBuffer += 16;

        case MODE_CFB1:
                iv = cipher->IV;
            for (i = numBlocks; i > 0; i--) {
                      memcpy(outBuffer, input, 16);
                for (k = 0; k < 128; k++) {
                            rijndaelEncrypt(key->ek, key->Nr, iv, block);
                    outBuffer[k >> 3] ^= (block[0] & 0x80U) >> (k & 7);
                    for (t = 0; t < 15; t++) {
                          iv[t] = (iv[t] << 1) | (iv[t + 1] >> 7);
                         iv[15] = (iv[15] << 1) | ((outBuffer[k >> 3] >> (7 - (k & 7))) & 1);
                outBuffer += 16;
                input += 16;

                return BAD_CIPHER_STATE;
          return 128*numBlocks;
    LVL 51

    Expert Comment

    hmm, my C knowledge is a bit rusty, but I guess you need to pass references for the blocks too, something like:
         blockEncrypt(&cipherInst, &keyInst, &inBlock, blockLength, &outBlock);

    Author Comment

    If I give as  blockEncrypt(&cipherInst, &keyInst, &inBlock, blockLength, &outBlock);
    compiler generates warning
    109: warning: passing arg 3 of `blockEncrypt' from incompatible pointer type
    encrypt.c:109: warning: passing arg 5 of `blockEncrypt' from incompatible pointer type

    inBlock and outBlock are AES_BYTE array
    where :
    typedef unsigned char   BYTE;
    typedef BYTE      AES_BYTE;

    Noting that the call the demo fn. works when a numerical input is given as key but doesnt work for alphanumeric input. Also if the initKey is hardcorded in alphanumeric then it works fine.
    LVL 51

    Expert Comment

    > .. but doesnt work for alphanumeric input. Also if the initKey is hardcorded in alphanumeric then it works fine.
    have you checked if the input contains non-printables?

    Author Comment

    No it doesnt even work for non-printables

    Author Comment

    Hi ahoffmann

    The problem was not in blockEncrypt function. the problem was with the makeKey function. The AES algorithm needed the correction which accepted till 'f'. Thanks anyways,
    I wonder how such silly things eat up a lot of time.


    Featured Post

    6 Surprising Benefits of Threat Intelligence

    All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

    Join & Write a Comment

    Many people tend to confuse the function of a virus with the one of adware, this misunderstanding of the basic of what each software is and how it operates causes users and organizations to take the wrong security measures that would protect them ag…
    The term "Bad USB" is a buzz word that is usually used when talking about attacks on computer systems that involve USB devices. In this article, I will show what possibilities modern windows systems (win8.x and win10) offer to fight these attacks wi…
    Internet Business Fax to Email Made Easy - With eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
    Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now