Python

Python is a widely used general-purpose, high-level programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than would be possible in other languages. Python supports multiple programming paradigms, including object-oriented, imperative and functional programming or procedural styles. It features a dynamic type system and automatic memory management and has a large and comprehensive set of standard libraries, including NumPy, SciPy, Django, PyQuery, and PyLibrary.

Share tech news, updates, or what's on your mind.

Sign up to Post

Hello Experts I have a Python task which involves Matching Schemas

Basically I need to modify the columns in the dataset df to match the provided schema and return the first three rows of the dataset

The steps that are to be are completed are as follows:

Write a function matchSchema(df) that achieves the following:
1. Converts column active to type Boolean
2. Creates the column ‘price’ by converting the column ‘counts’ to type Double and dividing by 100
3. Drops the ‘counts’ column
4. Returns the first three rows of the resulting Dataframe

I have attempted the task with the following script:

import numpy as np
import pandas as pd

df = pd.read_csv('D:\matchSchema.csv')

def matchSchema(df):
    df['active'] = df['active'].astype('bool')
    df['price'] = df['cents']/100
    df.drop('cents', axis=1, inplace=True)
    return df.head(3)

matchSchema(df)

However I'm failing to get the following set of results
Return Array of correct size
Return Array of rows with correnct number of rows
0
OWASP Proactive Controls
LVL 13
OWASP Proactive Controls

Learn the most important control and control categories that every architect and developer should include in their projects.

Hello Experts,

Can someone please help me write a function that will return an int of the number of partitions in a dataset.

I checked out the following site for assistance, but it didn't seem to help

https://stackoverflow.com/questions/32031270/function-to-return-number-partitions
cc2_sparkcontext.csv
0
Hi,
I need to do the Custom Object Detection in Deep learning using Tensorflow, i'm having the below queries -


1. Using Tensorflow model for custom object detection. Is there any better approach than this?
2. Versions: How to find stable versions of python, Tensorflow, CUDA, cudnn for custom object detection?
3. Currently i am using tensorflow v 1.14, python 3.6, cuda 9.0 and cudnn 7.0
4. How to make TFrecord?

I am using the  following commands

"python generate_tfrecord.py --csv_input=images\train_labels.csv --image_dir=images\train --output_path=train.record"

"python generate_tfrecord.py --csv_input=images\test_labels.csv --image_dir=images\test --output_path=test.record"

and getting error in many lines

 C:\ProgramData\Anaconda3\envs\Tensorflow2\lib\site-packages\tensorflow\python\framework\dtypes.py:525:  FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy,  it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)])


Please suggest any better approach for this, and help me sort this out....
0
I have a computer with an NVIDIA Rdx 2080 graphics card. I’m trying to run python using it. I’ve looked online for the different options. I prefer to run it with anaconda python.

1). The first option I tried was to install CUDA and run it with Visual studios. I’ve followed the instructions and videos online. https://developer.nvidia.com/how-to-cuda-python

I got it to the point where I installed it but then got stuck when trying to interface with visual studios.

2). Looking online I saw an alternative which is something about running CUDA directly on anaconda which doesn’t seem to requiring CUDA separately. https://docs.anaconda.com/anaconda/user-guide/tasks/gpu-packages/

If anyone has experience on which option is better and can guide me through installing it, I would greatly appreciate it.
0
Hi ,

Could please some help to write Recursive Query in Hive (or in Pyspark) .



CREATE   TABLE VSOLD_TESTCALC  AS (
     WITH
        RECURSIVE TEST
        (
          PURCHASENUMBER
          , EffDt
          , NetSOLD
          , NetPurchase --Nate
          , PostSeqNum
          , TransCode
          , Amt
          , TransDescription
          , rnum
          , t_flg
          , SpecPeriodFlg
          , MaxSOLDLimit
          )

         AS
        ( SELECT
           PURCHASENUMBER
         , StartDt                                 AS EffDt
         , CAST(SOLD - sell AS DECIMAL(18,4)) AS NetSOLD
         , CAST(Purchase - sell AS DECIMAL(18,4)) AS NetPurchase
         , renewal_seq       AS PostSeqNum
         , CAST(0 AS                INTEGER)       AS TransCode
         , CAST(0.0 AS              DECIMAL(18,2)) AS Amt
         , CAST('INIT_POSTITION' AS VARCHAR(400))  AS TransDescription
         , rnum
         , t_flg
         , SpecPeriodFlg
         , MaxSOLDLimit
        FROM
           TESTDB.SOLD_CurveInitPoint

        UNION ALL

        SELECT
           b.PURCHASENUMBER
         , b.EffDt
         ,NetSOLD 
         , NetPurchase 
         , b.PostSeqNum
         , b.TransCode
         , b.Amt
         , b.TransDescription END
         , b.rnum
         , b.t_flg
         , Flg
         , b.MaxSOLDLimit
        FROM
           TEST a
         , TESTDB.SOLD_Transactions_TESTCALC b
        WHERE
           b.rnum        = a.rnum+1
      

Open in new window

0
I want to print a single number totaling the instances of a certain word. I can only get this far where pandas prints each total of instances on each row in a new column.

import pandas as pd
import re
new = pd.read_csv("fileName")
search = "selectedText"
new["newColumn"] = new["ColumnToCheck"].str.count(search, re.I)
0
Hi,

I found the script on githib that has helped me extract CCTV footage from a Dahua CCTV recorder hard drive but it fails after extracting 31.5GB of data and I need to extract 2TB.

This is the first time I have installed Python and run a Python script.

The Script is

''' Python 3.7  for Dahua  by Dmytro Moisiuk '''
'''Script for recovering video from DVR Dahua DHI-HCVR4104HS-S2 
The following parameters should be changes at the end of the script:
f- image for analysis; 
blocksize-block size for analysis; 
quality - determine by byte with offset 0x1D in the integer from the beginning of the signature 0x44484156FD;
For all quality video use 'all' without quotation marks;
'''

import os
import sys
import hashlib
import re
import binascii


def Time_conv(Time):
    Time_List = ['Time']
    Time_List.append(hex(Time))
    c = int(Time_List[1][4],16)
    d = int(Time_List[1][5],16)
    b = int(Time_List[1][3],16)
    a = int(Time_List[1][2],16)
    d = int(Time_List[1][5],16)
    e = int(Time_List[1][6], 16)
    f = int(Time_List[1][7], 16)
    g = int(Time_List[1][8], 16)
    h = int(Time_List[1][9], 16)
    #b = int(b,16)
    #print(b)
    if (((c % 4)*8 + d // 2))<10:
        dd = '0'+str(((c % 4)*8 + d // 2))
    else:
        dd = str(((c % 4)*8 + d // 2))
    if ((b % 4)*4+c//4)<10:
        mm = '0'+str((b % 4)*4+c//4)
    else:
        mm = str((b % 4)*4+c//4)
    if ((a*4) + b//4) < 10:
        yy = '0' + str((a*4) + b//4)
    else:
        yy = str((a*4)

Open in new window

0
Hi All,
I need to do a document image template matching with the original document image. Is it possible using Artificial Intelligence or Machine Learning?

Basically, we will get a filled document image as an input. We need to match it with the already existing documents image based on template matching and return the identification of the matched document image.

Please help.

Thanks In advance.

Regards,
Kavita Choudhary.
0
I need to find a website where a rank beginner can start to learn python. https://www.w3schools.com/ was a very helpful resource to me learning html and CSS because it gave you code that you could modify and run and see the results immediately. But they don't seem to have anything like that for python, which is unfortunate because the only way I can learn anything is interactively.

Is there a site that will work that way for python?

Thanks,
John
0
How to check layers in HDF5 file in machine learning generated for Object detection using ImageAI.

We are getting below error message when trying to view :

ValueError: You are trying to load a weight file containing 0 layers into a model with 147 layers.

Please guide, how we can generate layers and view them.

Regards,
Rajni
0
OWASP: Threats Fundamentals
LVL 13
OWASP: Threats Fundamentals

Learn the top ten threats that are present in modern web-application development and how to protect your business from them.

Hi,

We have a python script that runs a algorithm based on input provided. We need to deploy the same in windows / web application.

How we can convert python script to dll or libraries that can be used in asp.net project ?

Regards,
Naveen Jain
0
i am uploading an image into fire-base and trying to get download URL from which i can send that image to the App.


i tried these two methods
path=imageBlob.public_url-
this returns https://firebasestorage.googleapis.com/v0/b/chain-app-f54e3.appspot.com/o/images%2Foutput 

but this URL is not working to send the the image to the app i want the firebase downloadURL which contains alt=media&token=980d472d-1639-470c-94e5-42f0635ce89c

Another method I tried:

storage.child("images/example").getDownloadURL(token) And in this method HOW TO GET THE TOKEN

Here is my code snippet:

storage.child("images/example").getDownloadURL(token)
 imagePath = "D:\\Chatbot\\output.png"
 imageBlob.upload_from_filename(imagePath)
 path=imageBlob.public_url


storage.child("images/example").getDownloadURL(token) in this method it is asking to pass the token but its not the Bearer or User token it is the token which is added in the link of image which we can get in firebase.


Please suggest,
Regards,
Naveen Jain
0
Hi

I want advice on how to take a video stream from an IP camera in python.
I have been trying this:import numpy as np
import cv2

cap = cv2.VideoCapture('rtsp://admin:admin@192.168.7.251')

while(True):

    ret, frame = cap.read()
    cv2.imshow('Stream IP camera opencv',frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

It won't connect to the IP camera, and is saying cannot connect to source.
0
Hi,

We are looking to detect Bar code from android device in a live / recorded video along with Object on which Barcode is printed using ML based object detection.

We are using Google Mobile Vision to detect the barcode, but not able to detect the object (Box / Paper / Camera Image / Laptop screen image ) on which it is printed.

Looking for help.

Thanks!
Naveen Jain
0
Hello, working on the webscrapping project, I encountered a problem with my chrome driver. I decided to download to most updated version and it is still giving the error.

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 77

Any clue on how to fix it? It was working perfectly fine before.
Thanks!
0
Hi,
I have written following Python code,But it's Not working. No results showing

with open('Test.csv') as csvfile:
	      readCSV=csv.reader(csvfile,delimiter=',')
	      print(readCSV)
	      for row in readCSV:
		      print(row)

Open in new window

Program executes on the Python 3.7.0 Shell
0
I used Tweepy and Python to export tweets related to the United States. I included in the original code how to handle errors using a sleep time suggested by Tweepy's guide. However, the code stops running after a couple of minutes.

This is the error: TweepError: Twitter error response: status code = 429

After doing research, I decided to included the wait_on_rate_limit to handle the connection error, but it also didn't work.

api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, compression=True)

Open in new window


This method gave the following error: TweepError: Failed to send request: ('Connection aborted.', OSError("(10054, 'WSAECONNRESET')"))

This is the original code:

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

def limit_handled(cursor):
    while True:
        try:
            yield cursor.next()
        except tweepy.RateLimitError:
            time.sleep(16 * 60)
        except StopIteration:
            return

api = tweepy.API(auth)
filepath = 'export_tweets.json'
count = 0

with open(filepath, 'w') as f:

    max_tweets = 20000
    query = '"united states" OR "united states of america"'
    tweets = tweepy.Cursor(api.search, q=query, tweet_mode='extended', lang='en').items(max_tweets)
    for tweet in limit_handled(tweets):
        f.write(jsonpickle.encode(tweet._json, unpicklable = False) + '\n')
        count += 1
    print(count)

Open in new window


I am expecting the output to be a total of 20,000 extracted tweets related to the query, but the code stops running. I am starting to learn Python and would appreciate any advice on how to handle this error.
0
How to make Windows executable file for my python-based code so that i use it in any windows system. And how to run this program auto after a fixed interval of time so that i get the live updated data from the website to the Excel file.
Code.txt
0
Hi,

I am trying to establish communication between HMI and PLC by using the Raspberry Pi. The existing setup between HMI directs to PLC via serial communication and protocol on Modbus RTU.

I am trying to put Raspberry Pi between HMI and PLC. Raspberry pi will be master serial to pull data from PLC via Serial Modbus RTU and be slave serial for HMI. HMI will be master to pull data from raspberry pi.

How I can achieve this? I am trying this to get some extra data from plc for monitoring purpose and didn't disturb the existing system.
0
Become a Certified Penetration Testing Engineer
LVL 13
Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

I am a professional stock market trader dealing with charts, Excel, Amibroker, Python-based automated trading system,3-4 monitor connected with my system. I want to purchase a new system. But little confused about choosing between Intel Xenon vs i9 vs i7 vs X series based processor. Which will be good for me?
0
Running into an error executing the following PS command:

Get-content C:\temp\hosts.txt | forEach-object ([Net.Dns]::GetHostbyAddress($_)}>>C:\\temp\results.txt

Open in new window

ERROR SEEN BELOW:
Get-Content C:\users\FQKNB\Tetration_IP_073019.txt| ForEach-Object {([system.net.dns]::GetHostByAddress($_)).hostname >> c:\users\FQKNB\hostname.txt}
Get-Content C:\users\FQKNB\Tetration_IP_073019.txt | ForEach-Object { ([Net.Dns]::GetHostByAddress($_)} >> c:\temp\results.txt
Get-Content c:\temp\hosts.txt | ForEach-Object { ([Net.Dns]::GetHostByAddress($_) } >> c:\temp\results.txt
At line:1 cgar:80
+...p\hosts.txt | forEach-object ([Net.Dns]::GetHostbyAddress($_)}>>c:\\t ...

Open in new window

1
BS4 web scraper can't get text values from named class in <div>

Hi I am trying to use Beautifulsoup to get a value from a web field. I can get normal <div> tags but I don't get any results when I try to parse data from a named class in a <div> tag. Can anyone tell me what I am doing wrong?

from urllib.request import urlopen as ureq
from bs4 import BeautifulSoup as soup
from requests import get
import urllib
from bs4 import BeautifulSoup
my_url = 'https://www.padmapper.com/apartments/toronto-on'
uClient = ureq(my_url)
page_html = uClient.read()
page_soup = soup(page_html, "html.parser")

list_item = page_soup.find_all("div", {"class":"ListItemMobile_text__2IIjF"})
for ls in list_item:
print(ls.get_text())
uClient.close()

Open in new window

0
I am attempting to scrape the following URL:

https://www.oddsportal.com/soccer/england/premier-league/liverpool-norwich-4IMoMG3q/

Using the Network tab in Chrome's developer tools, you can see there is an API that feeds the data into the website - which is in the form of a JSON - and looks as below. This data is what I am trying to scrape.

https://fb.oddsportal.com/feed/match/1-1-4IMoMG3q-5-2-yj1e3.dat?_=1562831112277

This is the code I am currently trying to scrape this with:

from selenium.webdriver.chrome.options import Options
import json
import urllib.parse
from time import time

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

# Access the initial webpage to create the info_dict (including the match_id, and hash)
driver.get('https://www.oddsportal.com/soccer/england/premier-league/liverpool-norwich-4IMoMG3q')
page = driver.page_source
info_dict = json.loads(page.split('var page = new PageEvent(')[-1].split(');')[0])
xhash = urllib.parse.unquote(info_dict['xhash'])
match_id = info_dict['id']

# Access to the feed URL based on the values from the info_dict
driver.get('http://fb.oddsportal.com/feed/match/1-1-{}-1-2-{}.dat?_={}'.format(match_id, xhash, int(round(time()*1000)) + 1000))
print(driver.page_source)

Open in new window


The URL is built up of 3 factors - the match_id, the hash, and epoch time in milliseconds. However, when I try to access this in Selenium, I get the following response:

globals.jsonpCallback('/feed/match/1-1-4IMoMG3q-1-2-yjb3a.dat?_=1562795864899', {'e':'404'})

Open in new window


Would really appreciate any help with this, as I don't really understand where I'm going wrong!
0
I am trying to use Python and Boto3 to report on installed patches on a instance, I am just using some simple code like that at the moment to test on a single instance. but when i try and run it i get a python error 'OverflowError: date value out of range'

import boto3
import datetime
client = boto3.client('ssm')

response = client.describe_instance_patches(
    InstanceId='Instance ID'
)

print response
0
Hi I'm having old legacy code (python2) , that creates a signature file for a digest it uses m2crypto

I'm also having a function, that verifies whether the digest and a signature file do match

Now I have to reimplement the signing code using cryptography instead of m2crypto (and switch to python3 lateron)
However the resulting signature must be verifiable with the old m2crypto code. (The code doing the verification runs on machines.
 that I cannot update now, the signing however has to be migrated to cryptography (and python3)

Attached I have a full self explaining example.
I guess the issue lies somewhere in the padding, which is not identical with the m2crypto and cryptography solution.  but I don't know how to proceed.



The old signing code:
def old_sign(digest):
    import M2Crypto
    rsa = M2Crypto.RSA.load_key("k.key")
    return rsa.sign(digest, "sha1")

Open in new window


The old verification code:
def old_check_signature(digest, signature):
    import M2Crypto
    rsa = M2Crypto.RSA.load_pub_key("k.pub")
    try:
        rsa.verify(digest, signature, algo="sha1")
        return True
    except M2Crypto.RSA.RSAError as exc:
        args = exc.args
        if len(args) < 1:
            raise
        return False

Open in new window



FIrst failing attempt of rewriting the signing code with cryptography:
def new_sign1(digest):
    from cryptography.hazmat.backends import default_backend
    from cryptography.hazmat.primitives import hashes, serialization
    from cryptography.hazmat.primitives.asymmetric import padding
    from cryptography.hazmat.primitives.asymmetric import rsa

    key_data = open("k.key").read()
    key =  serialization.load_pem_private_key(
        key_data, password=None, backend=default_backend())
    return key.sign(
        digest,
        padding.PSS(
            mgf=padding.MGF1(hashes.SHA1()),
            salt_length=padding.PSS.MAX_LENGTH,
            ),
        hashes.SHA1())

Open in new window


Second attempt to implement with cryptography:
def new_sign2(digest):
    from cryptography.hazmat.backends import default_backend
    from cryptography.hazmat.primitives import hashes, serialization
    from cryptography.hazmat.primitives.asymmetric import padding
    from cryptography.hazmat.primitives.asymmetric import rsa

    key_data = open("k.key").read()
    key =  serialization.load_pem_private_key(
        key_data, password=None, backend=default_backend())

    return key.sign(digest, padding.PKCS1v15(), hashes.SHA1())

Open in new window


And here the code to test (you just need a key file named k.key and a public key file named k.pub You can use any matching  pair or just download the attached files:
dig = "f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0"
sigs = [old_sign(dig), new_sign1(dig), new_sign2(dig)]
print([old_check_signature(dig, sig) for sig in sigs])

Open in new window

sign_issue.py
k.key
k.pub
0

Python

Python is a widely used general-purpose, high-level programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than would be possible in other languages. Python supports multiple programming paradigms, including object-oriented, imperative and functional programming or procedural styles. It features a dynamic type system and automatic memory management and has a large and comprehensive set of standard libraries, including NumPy, SciPy, Django, PyQuery, and PyLibrary.