Confidence Intervals in Python

from scipy import stats
import scipy as sp
import numpy as np

s = np.array([1,2,3,4,4,4,5,5,5,5,4,4,4,6,7,8])
n, min_max, mean, var, skew, kurt = stats.describe(s)
std=math.sqrt(var)

#note these are sample standard deviations 
#and sample variance values
#to get population values s.std() and s.var() will work


#The location (loc) keyword specifies the mean.
#The scale (scale) keyword specifies the standard deviation.

# We will assume a normal distribution
R = stats.norm.interval(0.05,loc=mean,scale=std)


>>> R
(4.33017855099411, 4.54482144900589)

Finding and Modifying Elements in a List

Finding Things in a List

1.To see check to see if something is in a list:

Heroes=['Thor','Iron Man','Black Widow','Hulk','Iron Man']
#Let's see if Hawkeye is on the list
if'Hawkeye' in Heroes:
   print 'He is on the list'
else:
   print 'No love for archers :('
>>> 
No love for archers :(
 

2.To find index of an element in a list by name:

Heroes=['Thor','Iron Man','Black Widow','Hulk','Iron Man']
#finds index of first occurrence of 'Iron Man'
print Heroes.index('Iron Man')
>>> 
1 

Modifying Things in a List

List Comprehensions: [f(x) for x in S if P(x)]

1. How to Map Functions to a Values in a List

#I want to create a list of doubled values
values = [2, 3, 9, 5]

# way 1
dblvals=[num*2 for num in values]
print dblvals
>>> 
[4, 6, 18, 10]

# way 2
dbl=lambda x:x*2
dblvals2=map(dbl,values)
print dblvals2
>>> 
[4, 6, 18, 10]

 

2. Filtering Elements in List

values = [2, 3, 9, 5]
#I want to create a list of values less than 8
less8vals=[num for num in values if num<8]
print less8vals
>>> 
[2, 3, 5]
 

3. Conditionally Replace Elements in List

stuff=[a if a==0 else 2 for a in [0,1,0,3]]
>>> stuff
[0, 2, 0, 2]
 

or we can move the conditional outside the list comprehension-

def check(x):
    if x>3 and x<5:
        return 'True'
    else:
        return 'False'
nums=[1,2,3,4,5,6]
mynums=[num for num in nums if check(num)=='True']
>>> mynums
[4]
 

4. Putting Lists Together and Taking them Apart

#Pulling out Column of Data
#------------------------------------------------
xyvals=[[1,200],[2,300],[3,400]]
xv=[coords[0] for coords in xyvals]
yv=[coords[1] for coords in xyvals]
>>> xv
[1, 2, 3]
>>> yv
[200, 300, 400]
>>> 


#Putting Together Columns of Data
#------------------------------------------------
newcoord=[[xv[i],yv[i]] for i in range(0,len(xv))]
>>> newcoord
[[1, 200], [2, 300], [3, 400]]
 

5. Deleting Items From A List

sets=[[1,'A'],[2,'Q'],[3,'C'],[4,'Q'],[5,'E'],[2,'A']]

#p: pair 
#pc: pair counter

pc=0
for p in sets:
    if p[1]=='Q':
        del sets[pc]
    pc=pc+1
>>> sets
[[1, 'A'], [3, 'C'], [5, 'E'], [2, 'A']]

Converting Data Forms

 

 

 

 

 

 

 

 

 

 

 

Here is how you convert between data types:

x=12.34
print int(x) #converts x to an integer
print float(x) # converts x to a float
print str(x) #converts x to a string
print type(x) #gives the data type of x
>>> 
12
12.34
12.34
type 'float'
>>>

Here is how you convert between container types:

import numpy as np
a_list=[1,2,3,4]
print a_list
#1.list into array
a_array=np.array(a_list)
print a_array
#2. array into list
a_list=list(a_array)
print a_list
#3. list and array to tuple
a_tuple=tuple(a_list)
print a_tuple
a_tuple=tuple(a_array)
print a_tuple

>>> 
[1, 2, 3, 4]
[1 2 3 4]
[1, 2, 3, 4]
(1, 2, 3, 4)
(1, 2, 3, 4)
>>> 

How to use Groupby in Python

1. The Basics

2. Making it Useful

3. When it Gets Complicated

The Basics

Honestly, most of the documentation for itertools makes me want to claw my eyes out. On that note, here is a step by step introduction to one of the more useful functions in intertools: groupby().

Lets say we have a list of pets...

Pets=['Rat','Owl','Owl','Rat','Toad',
'Newt','Cat','Owl','Toad']

We want to group this list of pets by kind to look like this:

Pets=[['Cat'],
['Newt'],
['Owl', 'Owl', 'Owl'],
['Rat', 'Rat'],
['Toad', 'Toad']]

To do this we will need to..
1 import itertools
2.sort the data (this is important)
3.give the categories of pets a name. I chose pet_type
4 call the groups of pets something. I chose pet_group

Now every time you loop through groupby(pets) we can print off the pet_type and the members of pet_group

from itertools import groupby
Pets=['Rat','Owl','Owl','Rat','Toad',
'Newt','Cat','Owl','Toad']
Pets=sorted(Pets)
for pet_type, pet_group in groupby(Pets):
   print pet_type
   print list(pet_group)

>>> 
Cat
['Cat']
Newt
['Newt']
Owl
['Owl', 'Owl', 'Owl']
Rat
['Rat', 'Rat']
Toad
['Toad', 'Toad']

Now notice something important: I have to convert pet_group to a list before
printing it. Otherwise, I end up printing a group object.

Making it Useful

groupby() is useful for two things:
1. identifying the unique categories in a list: making a list of all pet_type
2. partitioning the list by category: making a list of all pet_group

This is done in a simple way below:

from itertools import groupby
pet_types=[]
pet_groups=[]
Pets=['Rat','Owl','Owl','Rat','Toad','Newt',
'Cat','Owl','Toad']
Pets=sorted(Pets)
for pet_type, pet_group in groupby(Pets):
   pet_types.append(pet_type)
   pet_groups.append(list(pet_group))

Our list of unique categories...

>>> pet_types
['Cat', 'Newt', 'Owl', 'Rat', 'Toad']

Our partitioned list ....

>>> pet_groups
[['Cat'], ['Newt'], ['Owl', 'Owl', 'Owl'], ['Rat', 'Rat'], 
['Toad', 'Toad']]

When it Gets Complicated

Lets say instead of having a simple list you need to group by column. Say you need

[['Snape', 'Prof'],
['BuckBeak', 'creature'],
['Harry', 'student'],
['Ron', 'student'],
['Hermione', 'student']]

Converted to...

[
 [['Snape', 'Prof']],

 [['BuckBeak', 'creature']],

 [['Harry', 'student'], 
 ['Ron', 'student'], 
 ['Hermione', 'student']]
 ]

The groupby function can accept keys to group by, so to get it to group by a particular column...

from itertools import groupby

Hogwarts=[['Snape', 'Prof'],
['BuckBeak', 'creature'],
['Harry', 'student'],
['Ron', 'student'],
['Hermione', 'student']]

categories=[]
groups=[]

#write the key so the groupby function
#knows it is grouping by the second column

second=lambda x:x[1]

#run loop
for category , group in groupby(Hogwarts,key=second):
    categories.append(category)
    groups.append(list(group))

which will give you:

>>> groups
[[['Snape', 'Prof']],

 [['BuckBeak', 'creature']], 

[['Harry', 'student'],
 ['Ron', 'student'], 
['Hermione', 'student']]]

>>> categories
['Prof', 'creature', 'student']

The only catch is that you need to have the data presorted by the column you are grouping by. To see how to sort data by columns click: Sorting in Python

Write a CSV File

Reading CSV files

Python has a csv reader. It converts each line in the csv file into a list.

The cvs reader is able to interpret two default versions or 'dialects' of csv
1)comma
2)tab
If you have another type of delimiter you need to create a brand new dialect. Take the below csv file:

Creature;Legs;Eyes
Dog;4;2
Cyclops;2;1
Spider;8;6

We can read the above by:

Happy data munging.