input: Dogri.txt, English.txt

1. remove tralling space in english using notepad++
2. lowercase english using notepad++


3. tokenize dogri corpus using Dogri_tokenizer_1.ipnyb file
save to Dogri_tokenize.txt


4. tokenize english using MT_tokenize_NLTK_eng.py file
save to Eng_tokenize_nltk.txt
//cat English.txt  | ~/Desktop/satsahib/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en > Eng_tokenize.txt


5. concatination removal in english using MT_concatnaion_removal_eng.py
save to Eng_concat_remove.txt

6. Dogri digit removal
save to Dogri_digit_removal.txt


7. split data into train tune test using 1train_tune_test_file_creation.sh

8. clean corpus using shell scripl
~/Desktop/satsahib/mosesdecoder/scripts/training/clean-corpus-n.perl train dogri eng trainclean 1 40

~/Desktop/satsahib/mosesdecoder/scripts/training/clean-corpus-n.perl tune dogri eng tuneclean 1 40

~/Desktop/satsahib/mosesdecoder/scripts/training/clean-corpus-n.perl test dogri eng testclean 1 40


OUTPUT:
trainclean.dogri trainclean.hin
tuneclean.dogri tuneclean.hin
testclean.dogri testclean.hin


      35 1 preprocessing steps.txt
      26 1train_tune_test_file_creation.sh
   64121 CombinedAll_Dogri_N.txt
   64122 CombinedAll_Dogri.txt
   64121 CombinedAll_Hindi_N.txt
   64122 CombinedAll_Hindi.txt
    2787 test.dogri
    2787 test.hin
   58547 train.dogri
   58547 train.hin
    2787 tune.dogri
    2787 tune.hin

