python

SQLAlchemy update if unique key exists

by

From version 1.2 SQLAlchemy will support on_duplicate_key_update for MySQL There is also examples of how to use it: from sqlalchemy.dialects.mysql import insert insert_stmt = insert(my_table).values( id=’some_existing_id’, data=”inserted value”) on_duplicate_key_stmt = insert_stmt.on_duplicate_key_update( data=insert_stmt.values.data, status=”U” ) conn.execute(on_duplicate_key_stmt) From version 1.1 SQLAlchemy support on_conflict_do_update for PostgreSQL Examples: from sqlalchemy.dialects.postgresql import insert insert_stmt = insert(my_table).values( id=’some_existing_id’, data=”inserted value”) do_update_stmt … Read more

sort pandas dataframe based on list

by

The improved support for Categoricals in pandas version 0.15 allows you to do this easily: df[‘LSE_cat’] = pd.Categorical( df[‘LSE’], categories=[‘Oands’,’Wetnds’,’Develd’,’Cn’,’Soys’,’Otherg’,’Wht’], ordered=True ) df.sort(‘LSE_cat’) Out[5]: Region LSE North South LSE_cat 3 3 Oands -47.986764 -32.324991 Oands 2 2 Wetnds -38.480206 -46.089908 Wetnds 1 1 Develd -36.157025 -27.669988 Develd 0 0 Cn 33.330367 9.178917 Cn 5 5 … Read more

ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

by

This error message… ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine …implies that the initialization of a new WebBrowsing Session i.e. Firefox Browser session was aborted. An established connection was aborted by the software in your host machine As per your code attempt the error is clearly coming … Read more

Python type hint for (any) class

by

I’d recommend using a combination of TypeVar, to indicate that your self.o value could be any arbitrary type, and Type, in the following way: from typing import TypeVar, Type T = TypeVar(‘T’) class MyObj: def __init__(self, o: T) -> None: self.o = o def get_obj_class(self) -> Type[T]: return type(self.o) def accept_int_class(x: Type[int]) -> None: pass … Read more

How can I tear down a SparkSession and create a new one within one application?

by

Long story short, Spark (including PySpark) is not designed to handle multiple contexts in a single application. If you’re interested in JVM side of the story I would recommend reading SPARK-2243 (resolved as won’t fix). There is a number of design decisions made in PySpark which reflects that including, but not limited to a singleton … Read more

error code: 521