Solved

Accessing legacy database in Django

Posted on 2015-01-19
12
168 Views
Last Modified: 2015-01-21
Hi
I am just getting started with Django/Python/postgresql and have hit a problem. I have set up a project and within that project I have created an app called "daypass". I have created a models.py by using the following
python manage.py inspectdb > models.py

that created my models file and I think I have tidied it up. This is what it looks like now:
class Facilities(models.Model):
    name = models.TextField(db_column='Name', blank=True)  # Field name made lowercase.
    address = models.TextField(db_column='Address', blank=True)  # Field name made lowercase.
    city = models.TextField(db_column='City', blank=True)  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode', blank=True)  # Field name made lowercase.
    price = models.TextField(db_column='Price', blank=True)  # Field name made lowercase.
    primary_key = "true"

    class Meta:
        managed = False
        db_table = 'Facilities'


class DjangoMigrations(models.Model):
    id = models.IntegerField(primary_key=True)  # AutoField?
    app = models.CharField(max_length=255)
    name = models.CharField(max_length=255)
    applied = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'django_migrations'


class FacilitiesFacilities(models.Model):
    name = models.TextField(db_column='Name')  # Field name made lowercase.
    address = models.TextField(db_column='Address')  # Field name made lowercase.
    city = models.TextField(db_column='City')  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode')  # Field name made lowercase.
    price = models.TextField(db_column='Price')  # Field name made lowercase.
   
 class Meta:
        managed = False
        db_table = 'facilities_facilities'

although I am not sure what Facilitiesfacilities is - there is only one table, facilities in the DB that I want to use.

I then did:
python manage.py validate
and got no errors
and then did
python manage.py syncdb
although nothing happened

So I then went into Python in my PowerShell (windows) and tried to do the following, since I want to make sure I can access the record set that exists in the DB and I get this error:
from daypass.models import Facilities
where Facilities is the name of the table in the DB
PS C:\Users\nicka_000\documents\coding\websites_go_here\hideyhole> python
Python 2.7.9rc1 (default, Nov 26 2014, 09:19:41) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from daypass.models import Facilities
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "daypass\models.py", line 1
SyntaxError: Non-ASCII character '\xff' in file daypass\models.py on line 1, but no encoding declared; see http://python
.org/dev/peps/pep-0263/ for details
>>>

When I visit that link the information doesn't make a huge amount of sense to me. I have put these entries into the models.py file
 c:/python27/python
-*- coding: latin-1 -*-
but still get the error.

Any ideas on what else I need to do?

NB when creating a new table and populating through Django API I had to put the name of the app in the settings file but I haven't done that for this legacy DB. If I put "daypass" into the settings.py I get an error that it doesn't exist (the daypass directory doesn't have a settings file in it i.e. it wasn't generated when I created the app. Do I need to copy in manually?

thanks for your help
thanks
0
Comment
Question by:lz7cjc
  • 6
  • 6
12 Comments
 
LVL 16

Assisted Solution

by:gelonida
gelonida earned 500 total points
Comment Utility
I assume that you are using one of these advanced editors, that save text files with three leading binary bytes (being called a BOM)

You do not see them if you use an editor handling BOMs as it will hide these bytes automatically.

Python however does not handle BOM headers at the moment.

Try to find an option in your editor, which allows to save the file without a BOM header or if you don't find it
write a small python script, reading models.py, and rewriting it without the first three bytes (I could help you with this if you don't find the editor option)

If you're under linux you can use some basic unix commands to strip of the BOM.

What editor are you using?
0
 

Author Comment

by:lz7cjc
Comment Utility
Hi
thanks for getting in touch. I am using PyCharm - I have found encoding settings but not sure what I should be doing with it
Screenshot-2015-01-19-17.47.11.png
0
 
LVL 16

Assisted Solution

by:gelonida
gelonida earned 500 total points
Comment Utility
try to save for example as UTF without BOM.

However I'm surprised, that pycharm would save files with a BOM.


Could you try following from an interactive python shell or from a small script and post the output?
It is just to confirm, that you really have an issue with the first few bytes (the BOM) of your python file.
fname = "models.py" # replace this with the correct path to your models.py
fin = open(fname, 'rb')
first_bytes = fin.read(10)
print(' '.join("%02x" % ord(val) for val in first_bytes))

Open in new window

0
 

Author Comment

by:lz7cjc
Comment Utility
I can't see an option to save without BOM. I got as far as the second line on that script:
IOError: [Errno 22] invalid mode ('rb') or filename: 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhol
e\\daypass.models.py'
0
 
LVL 16

Expert Comment

by:gelonida
Comment Utility
The error message indicates, that the python file cannot be located or is not readable.

I guess it's just the file name:

It should probably be:
 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhole\\daypass\\models.py'

alternatvively just open a cmd window, cd to the directory containing the models.py file and start python from there.

Then the filename will just be models.py

If that fails as well I have another idea how to get the right filename
0
 

Author Comment

by:lz7cjc
Comment Utility
oops!
here is the output:
ff fe 23 00 20 00 54 00 68 00
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 16

Accepted Solution

by:
gelonida earned 500 total points
Comment Utility
Your file contains a BOM marker at the beginning of the file "ff fe"
From then on every second byte seems to be 00, which seems to indicate some encoding like UTF-16.
Python needs a file. without the
'ff fe' and with all the '00' removed.

As I don't know your PC nor Pycharm nor exactly what what happened.

One possible solution:
create a new file (with notepad / notepad+ or with pycharm).
just enter one line in the file: with for example
#!/usr/bin/env python

Open in new window


Save it,
check with the code of my previous script, that the output will NOT give some output containing "ff" or "fe".

If that worked out, then.

copy paste the contents of the original models.py over, remove any encoding marker lines like "-*- coding: latin-1 -*-",
remove any special characters.
I personally never include any non ASCII characters in source code in order to avoid problems when people with different editors and different setups try to edit and modify such a file.

Now save it and check with the code of my previous script whether the 'ff fe' disappeared.


Alternatively you could try to regenerate the file with
python manage.py inspectdb > models.py, check whether the generated file is already with the 'ff' 'fe' or every second byte '00' or not and reapply your modifications.
0
 

Author Comment

by:lz7cjc
Comment Utility
ok I'll get on and do that now. In the meantime, what option should I choose for encoding within PyCharm - i have taken a screen capture of some of the options
Screenshot-2015-01-20-17.28.20.png
0
 

Author Comment

by:lz7cjc
Comment Utility
here is the output:
 23 21 2f 75 73 72 2f 62 69 6e
0
 
LVL 16

Expert Comment

by:gelonida
Comment Utility
well that looks perfect

this is the string
"#!/usr/bin" in ASCII as python requires it.

could you try again to run your script with this new version of models.py?
0
 
LVL 16

Expert Comment

by:gelonida
Comment Utility
UTF-8 is a good option However only if Pycharm does not at the BOM bytes as well.
Give it a atry.

the head of the file should look

23 21 2f 75 73 72 2f 62 69 6e

If there's anything strange like "ff fe" at the beginning the BOM has been added and you will have problems again
0
 

Author Comment

by:lz7cjc
Comment Utility
Success, I believe :))
I took Pycharm out the picture and just used notepad and have managed to syncdb albeit with some warnings around boolean:
daypass.AuthUser.is_active: (1_6.W002) BooleanField does not have a default value.
        HINT: Django 1.6 changed the default value of BooleanField from False to None. See https://docs.djangoproject.co
m/en/1.6/ref/models/fields/#booleanfield for more information.

I'll investigate with Pycharm how I can avoid encoding issues and thanks for your help. I will no doubt have a new question as I go to the next section of the tutorial!
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
PHP Sort Order 5 53
Path to Python 9 36
Recieving large files from China 4 33
SQL to update characters in table column 6 25
Developer tools in browsers have been around for a while, yet they are still heavily underused by developers. Developers still fix html or CSS then refresh page to see effect, or they put alert or debugger in JavaScript and then try again and again …
Recently I spent hours debugging an issue in a Rails project where ActiveRecord was causing MySQL errors trying to create a User object of a class at the top level of a Single Table Inheritance model structure.  It turns out `.create` behaves differ…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
This video teaches viewers how to create their own website using cPanel and Wordpress. Tutorial walks users through how to set up their own domain name from tools like Domain Registrar, Hosting Account, and Wordpress. More specifically, the order in…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now