Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Accessing legacy database in Django

Posted on 2015-01-19
12
Medium Priority
?
184 Views
Last Modified: 2015-01-21
Hi
I am just getting started with Django/Python/postgresql and have hit a problem. I have set up a project and within that project I have created an app called "daypass". I have created a models.py by using the following
python manage.py inspectdb > models.py

that created my models file and I think I have tidied it up. This is what it looks like now:
class Facilities(models.Model):
    name = models.TextField(db_column='Name', blank=True)  # Field name made lowercase.
    address = models.TextField(db_column='Address', blank=True)  # Field name made lowercase.
    city = models.TextField(db_column='City', blank=True)  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode', blank=True)  # Field name made lowercase.
    price = models.TextField(db_column='Price', blank=True)  # Field name made lowercase.
    primary_key = "true"

    class Meta:
        managed = False
        db_table = 'Facilities'


class DjangoMigrations(models.Model):
    id = models.IntegerField(primary_key=True)  # AutoField?
    app = models.CharField(max_length=255)
    name = models.CharField(max_length=255)
    applied = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'django_migrations'


class FacilitiesFacilities(models.Model):
    name = models.TextField(db_column='Name')  # Field name made lowercase.
    address = models.TextField(db_column='Address')  # Field name made lowercase.
    city = models.TextField(db_column='City')  # Field name made lowercase.
    postcode = models.TextField(db_column='Postcode')  # Field name made lowercase.
    price = models.TextField(db_column='Price')  # Field name made lowercase.
   
 class Meta:
        managed = False
        db_table = 'facilities_facilities'

although I am not sure what Facilitiesfacilities is - there is only one table, facilities in the DB that I want to use.

I then did:
python manage.py validate
and got no errors
and then did
python manage.py syncdb
although nothing happened

So I then went into Python in my PowerShell (windows) and tried to do the following, since I want to make sure I can access the record set that exists in the DB and I get this error:
from daypass.models import Facilities
where Facilities is the name of the table in the DB
PS C:\Users\nicka_000\documents\coding\websites_go_here\hideyhole> python
Python 2.7.9rc1 (default, Nov 26 2014, 09:19:41) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from daypass.models import Facilities
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "daypass\models.py", line 1
SyntaxError: Non-ASCII character '\xff' in file daypass\models.py on line 1, but no encoding declared; see http://python
.org/dev/peps/pep-0263/ for details
>>>

When I visit that link the information doesn't make a huge amount of sense to me. I have put these entries into the models.py file
 c:/python27/python
-*- coding: latin-1 -*-
but still get the error.

Any ideas on what else I need to do?

NB when creating a new table and populating through Django API I had to put the name of the app in the settings file but I haven't done that for this legacy DB. If I put "daypass" into the settings.py I get an error that it doesn't exist (the daypass directory doesn't have a settings file in it i.e. it wasn't generated when I created the app. Do I need to copy in manually?

thanks for your help
thanks
0
Comment
Question by:lz7cjc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 6
12 Comments
 
LVL 17

Assisted Solution

by:gelonida
gelonida earned 2000 total points
ID: 40558082
I assume that you are using one of these advanced editors, that save text files with three leading binary bytes (being called a BOM)

You do not see them if you use an editor handling BOMs as it will hide these bytes automatically.

Python however does not handle BOM headers at the moment.

Try to find an option in your editor, which allows to save the file without a BOM header or if you don't find it
write a small python script, reading models.py, and rewriting it without the first three bytes (I could help you with this if you don't find the editor option)

If you're under linux you can use some basic unix commands to strip of the BOM.

What editor are you using?
0
 

Author Comment

by:lz7cjc
ID: 40558117
Hi
thanks for getting in touch. I am using PyCharm - I have found encoding settings but not sure what I should be doing with it
Screenshot-2015-01-19-17.47.11.png
0
 
LVL 17

Assisted Solution

by:gelonida
gelonida earned 2000 total points
ID: 40558156
try to save for example as UTF without BOM.

However I'm surprised, that pycharm would save files with a BOM.


Could you try following from an interactive python shell or from a small script and post the output?
It is just to confirm, that you really have an issue with the first few bytes (the BOM) of your python file.
fname = "models.py" # replace this with the correct path to your models.py
fin = open(fname, 'rb')
first_bytes = fin.read(10)
print(' '.join("%02x" % ord(val) for val in first_bytes))

Open in new window

0
Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

 

Author Comment

by:lz7cjc
ID: 40558263
I can't see an option to save without BOM. I got as far as the second line on that script:
IOError: [Errno 22] invalid mode ('rb') or filename: 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhol
e\\daypass.models.py'
0
 
LVL 17

Expert Comment

by:gelonida
ID: 40558560
The error message indicates, that the python file cannot be located or is not readable.

I guess it's just the file name:

It should probably be:
 'C:\\Users\nicka_000\\Documents\\Coding\\websites_go_here\\hideyhole\\daypass\\models.py'

alternatvively just open a cmd window, cd to the directory containing the models.py file and start python from there.

Then the filename will just be models.py

If that fails as well I have another idea how to get the right filename
0
 

Author Comment

by:lz7cjc
ID: 40559277
oops!
here is the output:
ff fe 23 00 20 00 54 00 68 00
0
 
LVL 17

Accepted Solution

by:
gelonida earned 2000 total points
ID: 40559668
Your file contains a BOM marker at the beginning of the file "ff fe"
From then on every second byte seems to be 00, which seems to indicate some encoding like UTF-16.
Python needs a file. without the
'ff fe' and with all the '00' removed.

As I don't know your PC nor Pycharm nor exactly what what happened.

One possible solution:
create a new file (with notepad / notepad+ or with pycharm).
just enter one line in the file: with for example
#!/usr/bin/env python

Open in new window


Save it,
check with the code of my previous script, that the output will NOT give some output containing "ff" or "fe".

If that worked out, then.

copy paste the contents of the original models.py over, remove any encoding marker lines like "-*- coding: latin-1 -*-",
remove any special characters.
I personally never include any non ASCII characters in source code in order to avoid problems when people with different editors and different setups try to edit and modify such a file.

Now save it and check with the code of my previous script whether the 'ff fe' disappeared.


Alternatively you could try to regenerate the file with
python manage.py inspectdb > models.py, check whether the generated file is already with the 'ff' 'fe' or every second byte '00' or not and reapply your modifications.
0
 

Author Comment

by:lz7cjc
ID: 40560226
ok I'll get on and do that now. In the meantime, what option should I choose for encoding within PyCharm - i have taken a screen capture of some of the options
Screenshot-2015-01-20-17.28.20.png
0
 

Author Comment

by:lz7cjc
ID: 40560236
here is the output:
 23 21 2f 75 73 72 2f 62 69 6e
0
 
LVL 17

Expert Comment

by:gelonida
ID: 40560669
well that looks perfect

this is the string
"#!/usr/bin" in ASCII as python requires it.

could you try again to run your script with this new version of models.py?
0
 
LVL 17

Expert Comment

by:gelonida
ID: 40560674
UTF-8 is a good option However only if Pycharm does not at the BOM bytes as well.
Give it a atry.

the head of the file should look

23 21 2f 75 73 72 2f 62 69 6e

If there's anything strange like "ff fe" at the beginning the BOM has been added and you will have problems again
0
 

Author Comment

by:lz7cjc
ID: 40561686
Success, I believe :))
I took Pycharm out the picture and just used notepad and have managed to syncdb albeit with some warnings around boolean:
daypass.AuthUser.is_active: (1_6.W002) BooleanField does not have a default value.
        HINT: Django 1.6 changed the default value of BooleanField from False to None. See https://docs.djangoproject.co
m/en/1.6/ref/models/fields/#booleanfield for more information.

I'll investigate with Pycharm how I can avoid encoding issues and thanks for your help. I will no doubt have a new question as I go to the next section of the tutorial!
0

Featured Post

Survive A High-Traffic Event with Percona

Your application or website rely on your database to deliver information about products and services to your customers. You can’t afford to have your database lose performance, lose availability or become unresponsive – even for just a few minutes.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developer tools in browsers have been around for a while, yet they are still heavily underused by developers. Developers still fix html or CSS then refresh page to see effect, or they put alert or debugger in JavaScript and then try again and again …
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question