1.In this problem and the next one, we’re going to make a very simple spam checker program by just looking
ooking at how likely a given email is to be spam based on the words it contains. In particular, in this problem we’re going to count how often words are present in spam emails within some set of training data (which here means a set of emails that have already been marked as spam or not spam manually).
We have already started to write a function spam_score(spam_file, not_file, word), which takes in two filenames, along with a target word (a lowercase string). Both filenames refer to text files which must be in the same directory as hw07.py (we’ve provided several such files in hw07files.zip). The text files contain one email per line (really just the subject line to keep things simple) - you can assume that these emails will be a series of words separated by spaces with no punctuation. The first file contains emails that have been identified as spam, the second contains emails that have been identified as not spam.
Since you haven’t learned File I/O yet, we’ve provided code that opens the two files and puts the data into two lists of strings (where each element is one line - that is, one email). You then must complete the function, so that it returns the spam score for the target word. The spam score is an integer representing the total number of times the target word occurs across all the spam emails, minus the total number of times the word occurs in not-spam emails. Convert all words to lowercase before counting, to ensure capitalization does not throw off the count.
You are asked to carry out a study on behalf of a business analytics specialised consultancy on a subsample
on a subsample of weekly data from Randall’s Supermarket, one of the biggest in the UK. Randall’s marketing management team wishes to identify trends and patterns in a sample of weekly data collected for a number of their loyalty cardholders during a 26-week period. The data includes information on the customers’ gender, age, shopping frequency per week and shopping basket price. Randall’s operates two different types of stores (convenient stores and superstores) but they also sell to customers via an online shopping platform. The collected data are from all three different types of stores. Finally, the data provides information on the consistency of the customer’s shopping basket regarding the type of products purchased. These can vary from value products, to brand as well as the supermarket’s own high-quality product series Randall’s Top. As a business analyst you are required to analyse those data, make any necessary modifications in order to determine whether for any single customer it is possible to predict the value of their shopping basket.
Randall’s marketing management team is only interested in identifying whether the spending of the potential customer will fall in one of three possible groups including:
• Low spender (shopping basket value of £25 or less)
• Medium Spender (shopping basket value between £25.01 and £70) and
• High spenders (shopping basket greater than £70)
For the purpose of your analysis you are provided with the data set Randall’s.xls. You have to decide, which method is appropriate to apply for the problem under consideration and undertake the necessary analysis. Once you have completed this analysis, write a report for the Randall’s marketing management team summarising your findings but also describing all necessary steps undertaken in the analysis. The manager is a competent business analyst himself/herself so the report can include technical terms, although you should not exceed five pages. Screenshots and supporting materials can be included in the appendix.
After completing your analysis, you should submit a report that consists of two parts. Part A being a non-technical summary of your findings and Part B a detailed report of the analysis undertaken with more details.
Part A: A short report for the Head of Randall’s Marketing Management (20 per cent). This should briefly explain the aim of the project, a clear summary and justification of the methods considered as well as an overview of the results.
Although, the Head of Randall’s Marketing Management team who will receive this summary is a competent business analytics practitioner, the majority of the other team members have little knowledge of statistical modelling and want to know nothing about the technical and statistical underpinning of the techniques used in this analysis. This report should be no more than two sides of A4 including graphs, tables, etc. In this report you should include all the objectives of this analysis, summary of data and results as well as your recommendations (if any).
Part B: A technical report on the various stages of the analysis (80 per cent).
The analysis should be carried out using the range of analytics tools discussed:
• SPSS Statistics
Ensure that the exercise references:
• Binary and multinomial logistic regression
• Linear vs Logistic regression
• Logit Model with odds Ratio
• Co-efficients and Chi Squared
• MLR co-efficients
• Assessing usefulness of MLR model
• Interpreting a model
• Assessing over-all model fit with Psuedo R-Squared measures
• Classification accuracy (Hit Ratio)
• Wald Statistic
• Odd ratio exp(B)
• Ratio of the probability of an event happening vs not happening
• Ratio of the odds after a unit change in the predictor to the original odds
• Residuals analysis
• Cook’s distance
• Adequacy (with variance inflation factor VIF and tolerance statistic)
• Outliers and influential points cannot just be removed. We need to check them (typo? – unusual data?)
• Check for multicollinearity
Write a short and concise report to explain the technical detail of what you have done for each step of the analysis.
The report should also cover the following information:
• Any type of analysis that might be useful and check whether the main assumptions behind the analyses do not hold or cannot be
• Give evidence of the understanding of the statistical tools that you are using. For example, comment on the model selection procedure and the coefficient interpretation, e.g. comment on the interpretation of the logistic regression coefficients if such a method is used and provide an example of
• Conclusions and explanation, in non-technical terms, of the main points