Tweeter Temp Check #2 - Taking Data to the Cleaners

Taking Data to the Cleaners by Ashur Baroutta


    As discussed in our previous submission, we are following up on our installation and setup process by starting to receive and clean our data. Given we are importing 100-1000 tweets all together, we need to remove unwanted elements from our text in order to get a better quality analysis of the tweets content.

    It is important to note why this is necessary. What we are aiming to do with sentiment analysis is mine the subjective textual information of a post and use natural language processing and machine learning libraries/functions to rank the sentiment of the given post. Working with twitter posts and replies, we will choose to clean the mined data of symbols/characters that we don't want affecting our analysis, an example of this would be a hashtag symbol. We are using our imported library and accessing its sub function to substitute certain characters/symbols with empty space so we can more accurately rank sentiment. The following is our written code to clean our mined posts of data we find unnecessary. 


    After cleaning our data, we can start analyzing each post and assessing the outputs. Admittedly, the information we are mining is subjective and as such our results are of a subjective nature, we would hesitate to say results absolutely depict the true sentiment/intent of a users post, though the insights have proven to be valuable. Computational methods are provided for us in our imported libraries that use ready made algorithms to rank sentiment for us. These are made by a collaboration of scientist in multiple fields. After making our function to clean the data we don't want, we want to format our output into something readable, the following is the code written for setting up our dataframe and integrating our subjectivity/polarity methods (topical related/negative, neutral/positive sentiment) from the TextBlob library. 




    Now that we've got our desired output format we can begin to analyze different outputs in our next submission. 





Comments

  1. EricaFebruary 11, 2022 at 11:02 AM

    Ashur,
    I love this project and completely understand why you're keeping it organized the way you are, just keep in mind you're on the hook for ~8-9 hours per week. 😉

    ReplyDelete

Post a Comment

Popular posts from this blog

IP #6 : Reversing an Integer

Image
 Reversing an Integer by Ashur Baroutta     The focus of this submission will be on reversing a given integer passed into our function. Our problem statement is as follows "Given a signed 32-bit integer x, return x with its digits reversed. If reversing x causes the value to go outside the signed 32-bit integer range [-2^31, 2^31 -1], then return 0." Seems simple, though was actually a little odd. My initial reaction is to convert the integer to a string and then just reverse each character. Though in a way that feels like cheating as I don't think that solution is in the spirit of the assignment,  and it takes more resources to complete.      Pivoting my attention to a mathematical solution made the problem a lot more fun. Say our first example for x is 123. The way we attack it is to setup variables to track the reversed total and to track the remainder of a given iteration, we also create a loop iterating through while x does not equal 0.  The ...
Read more

IP#5 : LinkedList Removing Duplicates II

Image
 Removing Duplicates II by Ashur Baroutta         In our last submissions, we worked on removing duplicates, this week we look at the more complicated problem of the same type. Our problem statement is " Given the  head  of a sorted linked list,  delete all nodes that have duplicate numbers, leaving only distinct numbers from the original list . Return  the linked list  sorted  as well ."      So not only do we have to remove duplicates, but we cannot return values that are not distinct, for example, if our list originally contained three 1's, we need to delete all the 1's in our list as even if we delete all but one of them, the number 1 itself is not a distinct value.      This took me about 3 days to think through and complete, the following is our code submission.  We start by first creating a new ListNode called newlist. We set its .next reference pointing to head (the ListNode passed ini...
Read more