Link to home
Start Free TrialLog in
Avatar of msidnam
msidnamFlag for United States of America

asked on

-bash: [: too many arguments. I get this error when ssh to a new Azure VM

I created a new server from the Azure Marketplace (CIS Hardened Ubuntu 18.04) and added a pub/private key that I've been using for other servers using the same image.

When I ssh to that new server I'm getting the error: "-bash: [: too many arguments"

I've tried to search that error but most of the results are for people making scripts in bash. not SSH to servers.
Avatar of omarfarid
omarfarid
Flag of United Arab Emirates image

Check the .profile or .bash_profile or similar files in /etc for wrong command or * character. This could be the issue.
Avatar of msidnam

ASKER

this is what I have in the .profile. I dont see anything out of the ordinary:

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
        . "$HOME/.bashrc"
    fi
fi

# set PATH so it includes user's private bin if it exists

if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi



I don't see any files with * or commands in the /etc directory.

I did setup another VM from Azure thats just ubuntu 18.04 (Not the CIS Hardened version) and that one does not give me the error when using ssh to access the server. Im wondering if Azure has a bad image?


What about the other files
~/.bash_profile or ~/.bash_login
How  do you ssh? Can you show the command?
Avatar of msidnam

ASKER

no bash_profile or .bash_login. Only a .bash_logout

We ssh in using local linux machines from home as well as putty for a windows workstation.

We also ssh from within other Azure VM's. The command we use is: ssh -i .ssh/privkey.pem username@ipaddress

We get the error using all three of these ways. But only on the new VM we created. the others (which have been up and running for over a year) don't give us those errors. If i do a uname -a, I do notice a different version.

VM not working : Linux <computer name> 5.4.0-1025-azure #25~18.04.1-Ubuntu SMP Sat Sep 5 15:28:57 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

VM working: Linux <computer name> 5.3.0-1022-azure #23~18.04.1-Ubuntu SMP Mon May 11 11:55:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The non working one seems to be a higher version. perhaps that image is having an issue?

Avatar of noci
noci

Filenames starting with a . are hidden from view.. (like the DOS/Windows HIDDEN file attribute).
ls -a make them visible.

file used in login:
/etc/profile
~/.profile
~/.bash_profile
~/.bash_login

Normal script running:
~/.bashrc
/etc/bash/bashrc   (+ /etc/bash/bashrc.d)

Then any command can be a bash script containing a problem...
Often /etc/profile uses a bunch of scipts in /etc/profile.d
Avatar of msidnam

ASKER

I went ahead and created two more VMs. One running Ubuntu 18.04 (Not CIS Hardened) and another one, CIS Hardened 16.04 (Not 18.04). Both of those work with no issues.

The error comes straight out of the box right after creating it. I have to think its something with that image, kernel, etc. We have not added anything to the server at all (scripts, updates, apps, etc) when this happens. 
This type of annoyance is dirt simple to debug.

Go through the files in noci's comment.

Place the following at top of each file...

echo ".bashrc" (or related file name)

set -x

Open in new window


Then login.

You'll have the filename output, then output will die in the file + line number where the problem occurs...

You'll have to do the same thing with any /etc/profile.d/* files, if you've modified these...
Avatar of msidnam

ASKER

Just to be clear. This happens right out of the box after I create the Azure VM. We do not modify or add anything to the server, unless the system does it itself during initial boot up since its a CIS version of Ubuntu. But that I don't know since i dont see the machine booting since its in Azure.

i did set -x on .bashrc and .profile but I dont see anywhere where it stops and tells me a line number.

This is what I get when logging on:

# Message of the DAY
Last login: Sat Sep 12 00:11:47 2020 from x.x.x.x
-bash: [: too many arguments


But its said that even before putting in set -x at the top of .bashrc and .profile.

Who is the root user of this system?
root may need to check the file /etc/profile which login bash will read and execute.

Also, can you use -vv with ssh command? This should give some debugging info.
Now you've provided enough information to debug the problem.

Context is everything.

Remove all the echo statements + set -x statements as the problem seems related to PAM, which is where MOTD (message of the day) output generation occurs...

Most likely somehow an error has crept into your MOTD generation, which is wildly different across Distros + also modified by various PAM modules installed.

One of the first actions I take on every new install is just to disable PAM, as it's rarely ever required.

If this is possible, add the following to your /etc/ssh/ssd_config file, then do a service sshd restart.

usePAM no
useDNS no

Open in new window


This makes ssh logins much faster also.
If you must debug PAM, then you'll do this...

# in one window
inotifywait -e OPEN -mrq /etc

# in another window login

Open in new window


This will show you an ordered trace of each file opened related to your login, likely /etc/pam.d/* or something similar.

Likely you'll instantly see what file name is involved.

If not, you'll just have to open each file + fix the problem.
Most common problem is a file missing #!/bin/bash in top line or having something else in top line, then BASH code inside the file.
Avatar of msidnam

ASKER

I went ahead and removed the MOTD for my login but as soon as I ssh I still get:

login as: user
Authorized uses only. All activity may be monitored and reported.
user@x.x.x.x's password:
-bash: [: too many arguments
user@dev-test:~$


I used this to disable the MOTD: 
 touch $HOME/.hushlogin

Open in new window

When I out usePAM no and useDNS no in the sshd_config file it didnt take it away.

I'm using a pub/priv key to login to ssh. Not sure if that has anything to do with it.
Can you try login with username and password and see?
Avatar of msidnam

ASKER

omarfarid,
Attached is what i get if i ssh using -vv
ssh_login.txt

This happens if i use a password as well. I created a new user with just a password and same thing. 
Do you have this dir /etc/init.d  ?

If yes, what you get if you run

ls -l /etc/update-motd.d

Also, are you setting your shell prompt (PS1) or any env variable while login?
Can you change the login shell for user in /etc/passwd to /usr/bin/bash -x ?

I never tried that, but you may try it for specific user for testing.
You cant add parameter in /etc/passwd.  you just try to use the  file  "/usr/bin/bash -x"    which doesn't exist.  So logon will fail.
Did you verify the /etc/bash_completion.d and ~/.bash_completion directories/scripts, it is also scanned for scripts.
You can run the following: strace bash -l to see what happens during logon... bash -l (el)  is how the login is run.
You said, "When I out usePAM no and useDNS no in the sshd_config file it didn't take it away."

Clarify what this means.

Maybe you got some error when you restarted sshd or maybe this seems to have no effect...

At least for sshd versions released over the past decade, sshd accepts these settings...

net17 # cat /etc/ssh/sshd_config | tail 
# http://linuxaria.com/howto/4-tricks-to-speed-up-ssh-connections
# AddressFamily inet
# http://www.azoft.com/spotlight/2011/04/18/ssh-ubuntu-100.html
# http://blog.tcs.de/speed-up-ssh-connections-with-compression-on-os-x/
# http://forums.gentoo.org/viewtopic-t-47746-view-next.html?sid=d4b05b81dcfe95b8e6f5313b06cd7b26
# touch ~/.hushlogin
PrintMotd yes
UsePAM no
UseDNS no
#GatewayPorts yes

net17 # service sshd restart

Open in new window


Notice the settings in the sshd config file, then restarting sshd works correctly.

Tip: You may have to resort to using inotifywait to trace file execution (explained above).

Sometimes the simple solution is seeing what files are executed.
Another debug trick...

# On your server open an ssh connection, then run a copy of sshd with maximum debug spew...
/usr/sbin/sshd -p 55555 -D -ddd -e

# Then initiate an ssh session with maximum debug spew...
ssh -vvvv -i /path-to-key $user@$host

Open in new window


I doubt this will help + work trying, if all else fails.

More than likely the inotifywait trick will work... and... likely also good to change the inotifywait command to also trace all user login files hit, which will be the following for the root user...

inotifywait -e OPEN -mrq /etc ~root

Open in new window


This will expand the inotifywait command to show how /etc system files + user login files interact... or run in sequence... related to each other...

The other trick is just to comment out everything in all your startup files, as the BASH error might be caused by some external program being run in some way.

Bottom line is you rule out everything, until the solution is all that's left.
Avatar of msidnam

ASKER

A lot of suggestions, thank you. I'll try to answer in order without sounding confusing.

omarfarid,
When I do ls -l /etc/update-motd.d  i get, total 0

noci,
I did look at directory  /etc/bash_completion.d  and it has a few files in the directory but when i view them I dont see anything odd, although I'm not sure what to look for.

When i do the strace command it gives me a lot of info that makes no sense to me and it wouldn't let me cntrl-c out of it.

David,
For UsePam no and UseDNs no, I added them to the end of the sshd_config file (note, when I dded the entries i had usePAM and useDNS with a lowercase U. The entries in the file had uppercase U), restarted sshd service and logged back in. I did not notice any change as I still got the error. Looking at your latest comment, I went back in the file and noticed that the file already has entries for them (based on your comment) UsePAM was set to yes I changed it to no. UseDNS was commented out I uncommented it, saved the file, restarted sshd, logged out and then back in and still the same thing (Note, I removed my entries i put in at the end of the file). I get the too many arguments error and thats it.

I tried doing the sshd command with a different port but but since this is a hardened CIS image i think i need to open the firewall or iptables in ubuntu and I am not familiar with that.

I did do the inotifywait on one session and then logged on to another session and I'll attach watch it shows when logging in. I see it hitting the directories/scripts that you all mention. But I still cant find out where the issue is.I'll attach watch displays when i login. I'm only attaching what i see immediately after putting in my password. It has a lot more stuff before logging in that i can attach as well.

I'm doing my best to give you all the info. I don't work on Linux much. I know enough to get around the OS and do some basic troubleshooting. apologies if my answers are not what you are expecting.

Also, remember/keep in mind that this happens immediately after I create the VM. I am not adding anything to the system, scripts, directories, programs, etc. I also do not know what the image changes during boot up. Since its a CIS Hardened Ubuntu image it very well could have scripts running when it first boots up to harden it but I dont know.All i know is right after Azure says that the resource is ready, i ssh to the newly created VM and get the too many arguments error. This VM is for our devlopers and the only thing they've told me (which alerted me to the issue) is this:

"Something to do with the initial shell loading seems to break variable substitution in a myriad of interesting ways. I think we should kill it and spawn another one."

"We moved past blockers with the /tmp directory being possibly mounted with `noexec` by explicitly setting the tmp dir for the Ruby orchestration framework capistrano. Then ssh keys. Now it's another root directory access being blocked and we were trying to workaround that"

Straight out of the gate the new VM has issues.


I appreciate your suggestions but I'm not sure if this is something that can be fixed. I really think the image from the Azure Marketplace is hosed.

inotifywait.txt

ASKER CERTIFIED SOLUTION
Avatar of noci
noci

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
The inotifywait file you posted is binary, so can't be reviewed.
Avatar of msidnam

ASKER

noci,
can i do strace bash -l 2>output.txt  from one session and then login using another session? will it still give me the data? or is strace bash -l 2>output.txt  only on that one session?
Avatar of msidnam

ASKER

David,
Try this one. Not sure what happened.

inotifywait.txt
Avatar of msidnam

ASKER

noci,
pushing it to a file was much easier to read and search. This is what i get:

write(2, "/etc/profile: line 22: [: too ma"..., 45/etc/profile: line 22: [: too many arguments
) = 45

when i view /etc/profile, line 22 is the one bolded:

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i

I'm not a programmer but I'm guessing the line of code is iterating through each of those .sh files in /etc/profile.d? I also notice, =45 but i dont know if thats another line number in the same file or a different file.

So its looking like one of those files in profile.d you all mentioned earlier but how do we determine which one?

Avatar of msidnam

ASKER

So i did a set -x on the /etc/profile (not the one inside my home dir) and got this:

+ for i in /etc/profile.d/*.sh
+ '[' -r /etc/profile.d/01-locale-fix.sh /etc/profile.d/apps-bin-path.sh /etc/profile.d/bash_completion.sh /etc/profile.d/cedilla-portuguese.sh '/etc/profile.d/*.sh' /etc/profile.d/tmout.sh /etc/profile.d/Z97-byobu.sh /etc/profile.d/Z99-cloudinit-warnings.sh /etc/profile.d/Z99-cloud-locale-test.sh ']'
-bash: [: too many arguments
+ for i in /etc/profile.d/*.sh
+ '[' -r /etc/profile.d/tmout.sh ']'
+ . /etc/profile.d/tmout.sh
++ TMOUT=900

I dont know if that means Z99-cloud-locale-test.sh  is the bad sh file or just happens to be the last one in the list. or maybe the whole line is a bad argument:

+ '[' -r /etc/profile.d/01-locale-fix.sh /etc/profile.d/apps-bin-path.sh /etc/profile.d/bash_completion.sh /etc/profile.d/cedilla-portuguese.sh '/etc/profile.d/*.sh' /etc/profile.d/tmout.sh /etc/profile.d/Z97-byobu.sh /etc/profile.d/Z99-cloudinit-warnings.sh /etc/profile.d/Z99-cloud-locale-test.sh ']'
I think issue is in

for i in /etc/profile.d/*.sh
Avatar of msidnam

ASKER

I commented out (in bold):

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "${PS1-}" ]; then
  if [ "${BASH-}" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi

#if [ -d /etc/profile.d ]; then
#  for i in /etc/profile.d/*.sh; do
#    if [ -r $i ]; then
#      . $i
#    fi

  done
  unset i
fi
umask 027

and got this:

-bash: /etc/profile: line 25: syntax error near unexpected token `done'
-bash: /etc/profile: line 25: `  done'



at least its a new error.
Avatar of msidnam

ASKER

So now i commented out the whole thing (sorry not a programmer didnt realize it was all the same line of code:

#if [ -d /etc/profile.d ]; then
#  for i in /etc/profile.d/*.sh; do
#    if [ -r $i ]; then
#      . $i
#    fi

# done
#  unset i
#fi

 and i no longer get the too many arguments.
You should comment done as well
You may check what you have for
etc/profile.d/*.sh
And if you have time and knowledge inspect each script
Avatar of msidnam

ASKER

omarfarid,
Thank you. I may have the developers who use this version of Ubuntu take a look since they may understand a little more the scripts and whats happening.

Thank you all for the suggestions and patience. I'm still getting used to this new answer system (i know its been around for a bit but I have no idea what happened to the points system) so please bear with me as I do my best to mark your answers and give you all credit.
Avatar of msidnam

ASKER

Thank you all again for the help. Much appreciated. 
The problem is probably a file with a space in the name in /etc/profile.d .. (that should not be there.... btw).

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r "$i" ]; then
      . "$i"
    fi
 done
  unset i
fi

Open in new window

The code is expected to be present as software that needs some stuff "preset" for a shell will drop a file in /etc/profile.d
(f.e. to setup aliases, of environment variables.., like EDITOR etc.).
Do your self a future favour and reinstall to the original code.

Then verify /etc/profile.d doesn't contain a file with a blank or '*' in the name!
That '*' or blank needs to go.

Be careful   rm -f * removes ALL files...   use rm -fi '*.sh'   the QUOTES are important  the -i also, only say Y on a file with '*' as name.
Avatar of msidnam

ASKER

noci,
You mentioned, "Do your self a future favour and reinstall to the original code."

We haven't changed any code in the VM. This happens right after we create it from the Azure marketplace.  
You said you modified /etc/profile... so undo those changes..., and add " aound $i  which will help here.
There is a reason for these files....
/etc/profile.d/01-locale-fix.sh /etc/profile.d/apps-bin-path.sh /etc/profile.d/bash_completion.sh /etc/profile.d/cedilla-portuguese.sh '/etc/profile.d/*.sh' /etc/profile.d/tmout.sh /etc/profile.d/Z97-byobu.sh /etc/profile.d/Z99-cloudinit-warnings.sh /etc/profile.d/Z99-cloud-locale-test.sh
to be present in that directory /etc/profile.d
Disabling them will not help with a next logon.

And THIS is the problematic one: '/etc/profile.d/*.sh'


Avatar of msidnam

ASKER

noci,
Gotcha, I understand what you are saying now.

This is what i show in the /etc/profile.d:

'*.sh'              Z99-cloud-locale-test.sh    bash_completion.sh
 01-locale-fix.sh   Z99-cloudinit-warnings.sh   cedilla-portuguese.sh
 Z97-byobu.sh       apps-bin-path.sh            tmout.sh

The two Z99 files are highlighted in green (not sure what that means). I don't see any spaces in the files, and one does have * in it, '*.sh', but in the /etc/profile file it does reference that *.sh file:

for i in /etc/profile.d/*.sh; do

I'm guessing its saying that to run all those scripts without having to reference each one?


No in the for statement it want a list of all file files ending in *.sh (* is wildcard for ANY character).
So it is not a needed file..
All the other files ARE needed though, and there is a reason for them to be run.
They are green because they are executable. which is not relevant.
Avatar of msidnam

ASKER

Apologies for the confusion. I am understanding it better now. I did what you mentioned and after logging in i do not get the error anymore.

I'll check with my developers to see if this also fixes the other issues they were having. Thank you.
Glad you got this fixed!

This type of problem can be a serious annoyance + time drain to fix.