quota

it is not clear from your post whether you have hard quota setting.

using root as the test account is not adviseable nor is it advisable to set quota on / versus creating a separate partition for /home

the other issue is you have two quota enforcement mechanisms: quota and grpquota.
root:other the group  might extend roots quota.

 create a new non admin user set their quota making sure it is a hard restriction.
then test the enforcement.

I created a user  test

[root@localhost ~]# quota test
Disk quotas for user test (uid 17073):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/fedora-root
                     28*     10      10   6days       9       0       0        
[root@localhost ~]# su test
[test@localhost root]$

Added test to sudoers file

[test@localhost root]$ sudo dd if=/dev/urandom of=/root/test.txt bs=32768 count=169
169+0 records in
169+0 records out
5537792 bytes (5.5 MB) copied, 0.487241 s, 11.4 MB/s

SHOULDN"T the above fail? Not sure what I am missing??

/etc/fstab has
/dev/mapper/fedora-root /                       ext4    defaults,usrquota,grpquota        1

Do I have to create seperate partition for home? How can I do that?

If /home is part of the same partition or in this case logical volume then no

Quotas are only able to restrict by Filesystem

I was suppose to write in /home/test/test.txt and now it worked .
It properly fails with quota exceeded

this worked
 dd if=/dev/urandom of=/home/test/test.txt bs=32768 count=169

THis is because "test" is the user

you have quota allocation per user and per group. test user is the one you setup with edquota username
edquota -g group see what options it lets you set here.
Note it is very dangerous to set a group quota where there are multiple users as that will enforced based on their combined space usage.

root may have one restriction, but the group other has none so at some point the group quota for other allows for root to have additional space beyond its own.