Solved

Accessing legacy database in Django

Posted on 2015-01-19
12
173 Views
Last Modified: 2015-01-21
Hi
I am just getting started with Django/Python/postgresql and have hit a problem. I have set up a project and within that project I have created an app called "daypass". I have created a models.py by using the following
python manage.py inspectdb > models.py

that created my models file and I think I have tidied it up. This is what it looks like now:
class Facilities(models.Model):
    name = models.TextField(db_column='Name', blank=True)  # Field name made lowercase.
    address = models.TextField(db_column='Address', blank=True)  # Field name made lowercase.
    city = models.TextField(db_column='City', blank=True)  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode', blank=True)  # Field name made lowercase.
    price = models.TextField(db_column='Price', blank=True)  # Field name made lowercase.
    primary_key = "true"

    class Meta:
        managed = False
        db_table = 'Facilities'


class DjangoMigrations(models.Model):
    id = models.IntegerField(primary_key=True)  # AutoField?
    app = models.CharField(max_length=255)
    name = models.CharField(max_length=255)
    applied = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'django_migrations'


class FacilitiesFacilities(models.Model):
    name = models.TextField(db_column='Name')  # Field name made lowercase.
    address = models.TextField(db_column='Address')  # Field name made lowercase.
    city = models.TextField(db_column='City')  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode')  # Field name made lowercase.
    price = models.TextField(db_column='Price')  # Field name made lowercase.
   
 class Meta:
        managed = False
        db_table = 'facilities_facilities'

although I am not sure what Facilitiesfacilities is - there is only one table, facilities in the DB that I want to use.

I then did:
python manage.py validate
and got no errors
and then did
python manage.py syncdb
although nothing happened

So I then went into Python in my PowerShell (windows) and tried to do the following, since I want to make sure I can access the record set that exists in the DB and I get this error:
from daypass.models import Facilities
where Facilities is the name of the table in the DB
PS C:\Users\nicka_000\documents\coding\websites_go_here\hideyhole> python
Python 2.7.9rc1 (default, Nov 26 2014, 09:19:41) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from daypass.models import Facilities
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "daypass\models.py", line 1
SyntaxError: Non-ASCII character '\xff' in file daypass\models.py on line 1, but no encoding declared; see http://python
.org/dev/peps/pep-0263/ for details
>>>

When I visit that link the information doesn't make a huge amount of sense to me. I have put these entries into the models.py file
 c:/python27/python
-*- coding: latin-1 -*-
but still get the error.

Any ideas on what else I need to do?

NB when creating a new table and populating through Django API I had to put the name of the app in the settings file but I haven't done that for this legacy DB. If I put "daypass" into the settings.py I get an error that it doesn't exist (the daypass directory doesn't have a settings file in it i.e. it wasn't generated when I created the app. Do I need to copy in manually?

thanks for your help
thanks
0
Comment
Question by:lz7cjc
  • 6
  • 6
12 Comments
 
LVL 16

Assisted Solution

by:gelonida
gelonida earned 500 total points
ID: 40558082
I assume that you are using one of these advanced editors, that save text files with three leading binary bytes (being called a BOM)

You do not see them if you use an editor handling BOMs as it will hide these bytes automatically.

Python however does not handle BOM headers at the moment.

Try to find an option in your editor, which allows to save the file without a BOM header or if you don't find it
write a small python script, reading models.py, and rewriting it without the first three bytes (I could help you with this if you don't find the editor option)

If you're under linux you can use some basic unix commands to strip of the BOM.

What editor are you using?
0
 

Author Comment

by:lz7cjc
ID: 40558117
Hi
thanks for getting in touch. I am using PyCharm - I have found encoding settings but not sure what I should be doing with it
Screenshot-2015-01-19-17.47.11.png
0
 
LVL 16

Assisted Solution

by:gelonida
gelonida earned 500 total points
ID: 40558156
try to save for example as UTF without BOM.

However I'm surprised, that pycharm would save files with a BOM.


Could you try following from an interactive python shell or from a small script and post the output?
It is just to confirm, that you really have an issue with the first few bytes (the BOM) of your python file.
fname = "models.py" # replace this with the correct path to your models.py
fin = open(fname, 'rb')
first_bytes = fin.read(10)
print(' '.join("%02x" % ord(val) for val in first_bytes))

Open in new window

0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 

Author Comment

by:lz7cjc
ID: 40558263
I can't see an option to save without BOM. I got as far as the second line on that script:
IOError: [Errno 22] invalid mode ('rb') or filename: 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhol
e\\daypass.models.py'
0
 
LVL 16

Expert Comment

by:gelonida
ID: 40558560
The error message indicates, that the python file cannot be located or is not readable.

I guess it's just the file name:

It should probably be:
 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhole\\daypass\\models.py'

alternatvively just open a cmd window, cd to the directory containing the models.py file and start python from there.

Then the filename will just be models.py

If that fails as well I have another idea how to get the right filename
0
 

Author Comment

by:lz7cjc
ID: 40559277
oops!
here is the output:
ff fe 23 00 20 00 54 00 68 00
0
 
LVL 16

Accepted Solution

by:
gelonida earned 500 total points
ID: 40559668
Your file contains a BOM marker at the beginning of the file "ff fe"
From then on every second byte seems to be 00, which seems to indicate some encoding like UTF-16.
Python needs a file. without the
'ff fe' and with all the '00' removed.

As I don't know your PC nor Pycharm nor exactly what what happened.

One possible solution:
create a new file (with notepad / notepad+ or with pycharm).
just enter one line in the file: with for example
#!/usr/bin/env python

Open in new window


Save it,
check with the code of my previous script, that the output will NOT give some output containing "ff" or "fe".

If that worked out, then.

copy paste the contents of the original models.py over, remove any encoding marker lines like "-*- coding: latin-1 -*-",
remove any special characters.
I personally never include any non ASCII characters in source code in order to avoid problems when people with different editors and different setups try to edit and modify such a file.

Now save it and check with the code of my previous script whether the 'ff fe' disappeared.


Alternatively you could try to regenerate the file with
python manage.py inspectdb > models.py, check whether the generated file is already with the 'ff' 'fe' or every second byte '00' or not and reapply your modifications.
0
 

Author Comment

by:lz7cjc
ID: 40560226
ok I'll get on and do that now. In the meantime, what option should I choose for encoding within PyCharm - i have taken a screen capture of some of the options
Screenshot-2015-01-20-17.28.20.png
0
 

Author Comment

by:lz7cjc
ID: 40560236
here is the output:
 23 21 2f 75 73 72 2f 62 69 6e
0
 
LVL 16

Expert Comment

by:gelonida
ID: 40560669
well that looks perfect

this is the string
"#!/usr/bin" in ASCII as python requires it.

could you try again to run your script with this new version of models.py?
0
 
LVL 16

Expert Comment

by:gelonida
ID: 40560674
UTF-8 is a good option However only if Pycharm does not at the BOM bytes as well.
Give it a atry.

the head of the file should look

23 21 2f 75 73 72 2f 62 69 6e

If there's anything strange like "ff fe" at the beginning the BOM has been added and you will have problems again
0
 

Author Comment

by:lz7cjc
ID: 40561686
Success, I believe :))
I took Pycharm out the picture and just used notepad and have managed to syncdb albeit with some warnings around boolean:
daypass.AuthUser.is_active: (1_6.W002) BooleanField does not have a default value.
        HINT: Django 1.6 changed the default value of BooleanField from False to None. See https://docs.djangoproject.co
m/en/1.6/ref/models/fields/#booleanfield for more information.

I'll investigate with Pycharm how I can avoid encoding issues and thanks for your help. I will no doubt have a new question as I go to the next section of the tutorial!
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Dictionaries contain key:value pairs. Which means a collection of tuples with an attribute name and an assigned value to it. The semicolon present in between each key and values and attribute with values are delimited with a comma.  In python we can…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question