Untangle or other linux dist. PPPOE Server

Roaring penguin hare a pppoe suite, (server & client).

http://www.roaringpenguin.com/pppoe/

The worst that can happen is that you can't find a package for your distro, in that case you can build from source.
Gentoo has rp-pppoe,
for RPM search the http://rpm.pbone.net/ for pppoe and equivalent RPM's surface.
for DEBIAN based: http://packages.debian.org/search?keywords=pppoe&searchon=names§ion=all

When you build from source, you need to toolchain: libtool, make, compilers etc. Which tools you need does depend on that source script. (A gentoo ebuild script does describe that steps involved in generating a package).

This is just a software based client i need a complete server solution and a modem hardware that should connect via cat6 line or cat5 or fiber line not the phone line

These are the programs that are installed from rp-pppoe.
So it's definitely NOT an client-only solution, program sinstall on my Gentoo system:

pppoe            pppoe-discovery  pppoe-server     pppoe-sniff      pppoe-status
pppoe-connect    pppoe-relay      pppoe-setup      pppoe-start      pppoe-stop

According to the man page is't NOT a highperformance server...

Here using Gentoo:
http://www.netup.biz/article-pppoe-server.php

Here Using Debian:
http://www.linuxforums.org/forum/servers/31548-pppoe-server.html#

Here they use the same tooling:
http://www.freeantennas.com/PPPoE-Server-HOWTO.html

So I beg to differ.

ohhh i see so what about the modem hardware that use a pppoe  connection over an cat5 line not a phone line ?

Please be more specific about the hardware....

CAT-5 would imply some sort of ethernet, or 4 or 8 wire connection.
If you mean you have RJ connectors (the flat (keyed?) 6-8 lead connectors).
8 pin connectors might also mean ISDN, ethernet or even regular phone
6 pin connectors are mostly used for POTS.

Here is some info about wiring....
http://en.wikipedia.org/wiki/Registered_jack

If you mean you need to setup another ppp connection then use the auto connect feature of pppd to connect on traffic. or pull up an other pppd line in the ifup script of your server connection.
But somewhat more info is needed to say anything about the modem you might need.

wrt. the bandwidth do you need to limit it or?

as you know standart adsl is routed to isps from phone line via dslam but we wont use phone line we plan to use standart cat 5 cable between dc and modems. So modem will connect over a cat5 to dc not phone line but we could not find a modem for this also fiber line should be ok instead of cat 5 so modem will connect the line via rj45 or fiber switch

Modems typically connect serial lines to phone lines. (ADSL is a variant of ATM).
DSLAM are ADSL modems combined in a box (more or less).

So you want ethernet to? regular phone/adsl/...? to ethernet

In that case why not use an ethernet bridge? Like...
http://www.patton.com/products/pe_products.asp?category=238

There will be sth. like on the image given below. there won't be any phone line we plan to connect modem to directly to the PPPOE server with cat5 or fiber cable. Do you see ? there is no phone line
Untitled.png

OK,
But how are the home computers connected to your modem.

modem is in the home . And they are connecting with standart cat5 line to the modem. Every thing in the scenario same with normal adsl line the only difference is we plan to use cat6 and fiber between dc and homes instead of phone line so we need a modem which has fiber optic input instead of phone line

Ok, A fibre optic transceivers then.

Like these:
http://www.dlink.ca/DE-854
http://www.ciao.co.uk/Repeaters_Transceivers_5220768_4-black_box-ethernet-fibre_optic-transceiver
http://www.irtelectronics.com/products/EOT-4580.html
http://www.motorola.com/Business/PK-EN/Business+Product+and+Services/Accessories/WLAN+Accessories/Power+Over+Ethernet/ES3000+Fiber+Optic+Transceiver_Loc%253AXU-EN%252CXN-EN%252CXC-EN%252CXM-EN%252CXE-EN%252CPK-EN%252CXF-EN
(just a random search)

This describe some specs,
http://www.globalspec.com/learnmore/optics_optical_components/fiber_optics/fiber_optic_transceivers

These are sometimes also called GBIC's.
Now if you use these you don't need pppoe, you could use IPSEC + XAUTH to prevent overly levels encasulation.